Wireless communication device and method for use therewith

ABSTRACT

An image of at least one object ( 14 ) is captured. An object class selection is received from a menu ( 70 ) of a plurality of object classes. At least one task menu specific to the object class selection is provided. A task selection is received from the at least one task menu. A task associated with the at least one object ( 14 ) is facilitated based on the image and the task selection.

RELATED APPLICATIONS

The present invention claims priority under 35 U.S.C. 120 as acontinuation of, “METHOD AND SYSTEM FOR FACILITATING TASKS USING IMAGESAND SELECTIONS FROM OBJECT CLASS AND TASK MENUS, filed on Oct. 27, 2000,having Ser. No. 09/698,310 that itself is a continuation in part of thefollowing applications having the same assignee as the presentapplication:

“ONLINE BIDDING METHODS AND SYSTEMS FOR AGGREGATE ORDERS”, filed Aug.14, 2000, having Ser. No. 09/638,815.

“METHODS AND SYSTEMS FOR FACILITATING FULFILLMENT OF AN AGGREGATE ORDERAT MULTIPLE RETAIL STORES”, filed Aug. 14, 2000, having Ser. No.09/638,470; and

“COMPUTER NETWORK NAVIGATION USING OBJECTS ABSENT OF OPTICAL CODES”,filed Aug. 18, 2000, having Ser. No. 09/641,207.

TECHNICAL FIELD

The present invention relates to wireless communication devices thatinclude image capture devices.

BACKGROUND OF THE INVENTION

Two commonly-used techniques for navigating to destinations of acomputer network, such as the Internet, include: (a) manually entering aUniform Resource Locator (URL) into a command line of a Web browser, and(b) clicking on a hyperlink displayed by a Web browser. Manuallyentering URLs having a long sequence of characters may be undesirablefor some end users.

Various recently-proposed techniques for navigating to destinations ofthe Internet use bar codes to obviate manual entering of URLs. U.S. Pat.Nos. 5,930,767, 5,936,726, 5,940,595, 5,969,324, 5,986,651, 5,995,105,6,032,195 and 6,081,827, which are assigned to Motorola, Inc. and arehereby incorporated by reference into the present disclosure, disclosevarious approaches to computer network navigation using bar codes.

U.S. Pat. No. 5,978,773 to Hudetz et al. discloses that a bar code usedto identify a product (e.g. a Universal Product Code) also may be usedto access remote computers on a network. To access an Internet resourcerelating to a particular product, a user enters the product's UPC. Adatabase retrieves the URL (Uniform Resource Locator) corresponding tothe UPC. The URL is used to access the Internet resource.

U.S. Pat. No. 5,804,803 to Cragun et al. discloses a method whichincludes determining whether or not a code is a UPC. If the code is aUPC, URL information stored in a product database is requested using theUPC as an index. If the code is not a UPC, the code is considered as aURL string which may or may not be in an abbreviated form.

International Publication No. WO 00/16211 to Digitalconvergence.com,Inc. discloses a method for interfacing scanned product information withthe manufacturer of the product over a global communication network. Aproduct identifier is accessed at a source location on the network.Routing information is appended to the accessed product identifier. Therouting information defines a remote location on the network. Theproduct information is transmitted from the source location to theremote location in accordance with the appended routing information.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is pointed out with particularity in the appended claims.However, other features of the invention will become more apparent andthe invention will be best understood by referring to the followingdetailed description in conjunction with the accompanying drawings inwhich:

FIG. 1 is a flow chart of an embodiment of a method of facilitatingtasks using images and selections from object class and task menus;

FIG. 2 is a block diagram of an embodiment of a system for facilitatingtasks using images and selections from object class and task menus;

FIG. 3 illustrates an embodiment of a hierarchy of object class and taskmenus;

FIG. 4 illustrates an embodiment of a network access apparatus providingan object class menu;

FIG. 5 illustrates an embodiment of the network access apparatusproviding a first task menu for a product class;

FIG. 6 illustrates an embodiment of the network access apparatusproviding a portion of a second task menu based on a selection from thefirst task menu;

FIG. 7 illustrates an embodiment of the network access apparatusproviding another portion of the second task menu; and

FIG. 8 illustrates an embodiment of the network access apparatusproviding information based on a selection from the second task menu.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

There is a heed for methods and systems which do not require an opticalcode, such as bar code, to be present on an object in order to opticallyuse the object to facilitate a task. To meet this need and other needs,embodiments of methods and systems for facilitating tasks using imagesand selections from object class and task menus are disclosed herein.Embodiments are described with reference to FIG. 1, which shows a flowchart of an embodiment of a method of facilitating tasks using imagesand selections from object class and task menus, and FIG. 2, which showsa block diagram of an embodiment of a system for facilitating tasksusing images and selections from object class and task menus.

As indicated by block 10, the method comprises capturing an image of ascene 12 which comprises at least one object 14. An end user may capturethe image using a digital camera 16 associated with a network accessapparatus 20. The digital camera 16 may comprise various image capturingdevices known in the art, such as a charge-coupled device (CCD) array orcomplementary metal-oxide semiconductor (CMOS) sensors, for example. Thedigital camera 16 may include a single CCD array to capture atwo-dimensional image of the scene 12. Alternatively, the digital camera16 may include two CCD arrays to capture a 3-D stereoscopic image of thescene 12. The image may be a still image or one of an animated sequenceof images.

The network access apparatus 20 may have various implementations. Absentthe digital camera 16, examples of the network access apparatus 20include, but are not limited to, a computer, a personal digitalassistant, a set-top box, a Web telephone, a two-way pager, a gameplayer, various wireless network access apparatus, a wireless wristcommunication device, a pen-shaped wireless communication device and acard-shaped wireless communication device. The digital camera 16 may beeither integrated with, attached to, or otherwise in communication with(e.g. via a memory card) any of the aforementioned apparatus. Ofparticular interest is the network access apparatus 20 comprising aWeb-enabled, wireless handheld device, such as a Web-enabled cellulartelephone, having an integrated digital camera.

As indicated by block 22, the method optionally comprises displaying theimage for view by the end user. The image is displayed by a displaydevice 24 responsive to the digital camera 16 via a processor 26.Preferably, the display device 24 comprises a liquid crystal displayhaving a pixel resolution, a number of pixel intensity levels and/or anumber of pixel colors sufficient for displaying the image.Alternatively, other display devices, such as a cathode ray tube, acomputer monitor, a television, a field emission display, a display on awireless telephone, a display on a wireless wrist communication device,a display on a pen-shaped wireless communication device, and a displayon a card-shaped wireless communication device may be employed as onehaving ordinary skill in the art will recognize.

The processor 26 either directs or assists in directing acts performedby various components of the network access apparatus 20. The processor26 may comprise: one or more microprocessors, one or moremicrocontrollers, or other implementations of a logic circuit. Theprocessor 26 may be directed by computer-readable content of acomputer-readable medium. Examples of the computer-readable mediuminclude a computer-readable storage medium and a computer-readablecommunication medium. Examples of computer-readable storage mediainclude, but are not limited to, a computer memory, a solid state memorycard, a flash memory, a magnetic floppy disk, a magnetic hard disk, andan optical disk. Examples of computer-readable communication mediainclude, but are not limited to, an electronic medium, an optical mediumand an electromagnetic medium.

As indicated by block 30, the method optionally comprises receiving oneor more signals indicating one or more user-selected portions of theimage. The end user selects the portion(s) of the image to contain oneor more of the at least one object 14, and to exclude at least anotherone of the at least one object 14. The one or more of the at least oneobject 14 in the user-selected portion(s) include those for which theuser desires to perform an associated task.

To provide the aforementioned signal (s), the network access apparatus20 comprises a user input interface 32. The user input interface 32senses one or more user-initiated actions which indicate theuser-selected portion(s) of the image, and generates the signal(s) basedthereon. Preferably, the user input interface 32 comprises atouch-sensitive screen of the display device 24. In this case, the enduser may use a stylus or a finger to draw one or more closed loopsaround one or more of the at least one object 14 to make a selectionthereof. Either in addition to or as an alternative to thetouch-sensitive screen, the user input interface 32 may comprise anycombination of a telephone keypad, one or more buttons, a mouse, atrackball, a pointing stick, a touch pad, a joystick, a keyboard, and avoice input device. The end user may select the portion(s) of the imageusing any of the aforementioned input devices. Using a voice inputdevice, for example, a user may speak a name and/or a description of theat least one object 14 to make a selection thereof.

As indicated by block 34, the method comprises providing a menu of aplurality of object classes. The plurality of object classes in the menumay comprise a person class, a product class, a company class, alocation class, an event class, or any combination thereof. The menu maybe provided locally by a memory 36 and/or remotely by a server system 40via a computer network 42. The menu is displayed by the display device24.

If the menu is provided remotely, a transceiver 44 may be directed bythe processor 26 to facilitate navigation to the server system 40 viathe computer network 42. Thereafter, the transceiver 44 may be directedby the processor 26 to receive the menu from the server system 40 viathe computer network 42. Examples of the transceiver 44 include, but arenot limited to, a wireless radio frequency transceiver, a wirelessmodem, a dial-up modem, a cable modem and a network interface card.Examples of the computer network 42 include, but are not limited to, aninternet, an intranet and an extranet.

As indicated by block 46, the method comprises receiving an object classselection from the menu. The object class selection is made by the enduser using the user input interface 32.

As indicated by block 50, the method comprises providing at least onetask menu specific to the object class selection. Each task menuprovides a plurality of tasks specific to a class of objects. Specificexamples of task menus are subsequently described with reference to FIG.3. The at least one task menu may be provided locally by the memory 36and/or remotely by the server system 40 via the computer network 42. Theat least one task menu is displayed by the display device 24.

As indicated by block 52, the method comprises receiving a taskselection from the at least one task menu. The task selection is made bythe end user using the user input interface 32.

As indicated by block 54, the method comprises facilitating a taskassociated with the at least one object 14 based on the image and thetask selection. The image and the task selection may be processedremotely by the server system 40 to facilitate the task. In this case,the transceiver 44 is used to communicate the task selection and atleast part of the image to the server system 40 via the computer network42. The at least part of the image may consist of either the entireimage or the user-selected portion(s) of the image. Optionally, theentire image is provided to the server system 40 along with dataindicating the user-selected portion(s).

The server system 40 provides an object recognizer 56 capable ofrecognizing objects absent of an optical code in a received image.Preferably, the object recognizer 56 is capable of recognizing objectsabsent of an optical code and objects having an optical code which isnot present in the received image. For example, the object recognizer 56may be capable of recognizing objects absent of either a one-dimensionalor a two-dimensional bar code, and objects having either aone-dimensional or a two-dimensional bar code which is not present inthe received image. Various known object recognition methodologies maybe employed by the object recognizer 56, as those having ordinary skillin the art of image processing will recognize.

Preferably, the object recognizer 56 is capable of recognizing bothsubstantially planar objects and substantially non-planar objects.Examples of substantially planar objects include, but are not limitedto, fingerprints, a printed page in a publication, a printed poster, anda printed sign. Examples of substantially non-planar objects include,but are not limited to, human faces, products in visually recognizablepackages and/or containers (e.g. cans, bottles, boxes and bags),visually-recognizable products themselves, landmarks and buildings.

It is further preferred that the object recognizer 56 be capable ofrecognizing objects independent of their absolute and relative locationsin the received image. Further, the object recognizer 56 preferably iscapable of recognizing objects independent of their orientation and/orsize in the received linage.

The object recognizer 56 may use a user-spoken or otherwise user-enteredname and/or description of the an object to assist in its recognitionand/or selection. For example, the end user may say “battery” to selecta battery within an image of a plurality of objects. The objectrecognizer 56, in turn, would attempt to recognize a battery within theimage.

The server system 40 further provides a task facilitator 60 responsiveto the object recognizer 56. The task facilitator 60 facilitatesperforming a task associated with the task selection and at least onerecognized object in the image. The task facilitator 60 either mayfacilitate a task for a single recognized object, or may facilitate arespective task for each of a plurality of recognized objects.

Each task may be at least partially facilitated using an associatedresource of the computer network 42. For example, one of the tasks inthe task menu may be facilitated for the at least one object 14 using acomputer network resource 61.

The task facilitator 60 may translate a combination of a recognizedobject and a task selection to an associated computer address of theassociated resource. The task facilitator 60 may comprise either adatabase or an associative memory which associates a respective computeraddress to each of a plurality of different object/task selectioncombinations. Examples of each computer address include, but are notlimited to, a URL, a uniform resource name (URN), a uniform resourceidentifier (URI) and an Internet protocol (IP) address. Potentialcomputer network addresses include all of those available via the WorldWide Web.

Optionally, the server system 40 provides an optical code detector 62capable of detecting and decoding an optical code in a received image.In this case, the task facilitator 60 is responsive to the optical codedetector 62 to facilitate a task based on an optical code/task selectioncombination. Examples of the optical code include one-dimensional andtwo-dimensional bar codes. The optical code may comprise a product codesuch as either a UPC bar code, an EAN (European Article Numbering) barcode; a JAN (Japanese Article Numbering) bar code or ISBN (InternationalStandard Book Number) bar code. The task facilitator 60 may compriseeither a database or an associative memory which associates a respectivecomputer address to each of a plurality of different optical code/taskselection combinations. Potential computer addresses resources includeall of those available via the World Wide Web.

Optionally, the task facilitator 60 also associates a respectivesoftware agent with each object/task selection combination and eachoptical code/task selection combination. Each software agent is usableto assist in performing an associated task. Alternatively, the computernetwork resource associated with a combination may provide a softwareagent.

Using the aforementioned components, the server system 40 may facilitatethe task as follows. The server system 40 receives the task selectionand the at least part of the image via the computer network 42. As usedherein, the phrase “received image” is meant, to indicate either anentire image or the user-selected portion(s) of the entire imagereceived by the server system 40. In other words, if an image isreceived along with data indicating the user-selected portion(s) of theimage, the “received image” indicates the user-selected portion(s) ofthe image.

The optional optical code detector 62 serves to determine if an opticalcode is present in the received image. If so, the task facilitator 60determines one or more computer addresses associated with one or morecombinations of the task selection with one or more optical codesdetected in the received image.

The object recognizer 56 serves to attempt to recognize one or moreobjects absent an optical code in the received image. Preferably, theone or more objects absent an optical code are recognized without use ofoptical character recognition. The task facilitator 60 determines one ofmore computer addresses associated with one or more combinations of thetask selection and one or more objects recognized in the received image.

The server system 40 may communicate each of the computer addressesdetermined by the task facilitator 60 to the network access apparatus20. This act may comprise communicating a single computer address eitherif one optical code is detected in the received image and no objects arerecognized in the received image, or if one object is recognized in thereceived image and no optical codes are detected in the received image.Alternatively, this act may comprise communicating a plurality ofcomputer addresses either if a plurality of optical codes are detectedin the received image, if a plurality of objects are recognized in thereceived image, or at least one optical code is detected in the receivedimage and at least one object is recognized in the received image.

The processor 26 directs the transceiver 44 to receive one or morecomputer addresses communicated by the server system 40 via the computernetwork 42. To assist in facilitating a task, the processor 26 directsnavigation to a computer network resource using a received computeraddress. This act may include automatically using a computer address tosubstantially immediately facilitate computer navigation to a computernetwork resource. Alternatively, this act may include storing one ormore computer addresses to facilitate a subsequent, delayed computernavigation to one or more computer network resources.

The transceiver 44 may send one or more messages to attempt to link to aserver which provides the computer network resource, and to request thecomputer network resource. Further, the transceiver 44 may receive therequested computer network resource in the form of one or more responsemessages from the server. Media content received from the computernetwork resource may be processed by the processor 26 based on markupdata received from the computer network resource. The processor 26provides a display signal to the display device 24 to display themarked-up content. The requested computer network resource may be a Webpage, for example, having visible-content and/or audible content.

Acts performed by the aforementioned components of the server system 40may be directed by computer-readable content of a computer-readablemedium. Examples of the computer-readable medium include acomputer-readable storage medium and a computer-readable communicationmedium. Examples of computer-readable storage media include, but are notlimited to, a computer memory, a magnetic floppy disk, a magnetic harddisk, and an optical disk. Examples of computer-readable communicationmedia include, but are not limited to, an electronic medium, an opticalmedium and an electromagnetic medium. The server system 40 may compriseeither a single server computer or a plurality of computers to providethe aforementioned components.

As an alternative to using the server system 40, the image and the taskselection may be processed locally by the network access apparatus 20 tofacilitate the task. In this case, the processor 26 is programmed tolocally perform the herein-described acts of the object recognizer 56,the task facilitator 60, and the optical code detector 62. In this case,the software agents for the tasks may reside in the memory 36.Optionally, the memory 36 comprises a memory card or an alternativemedium which is removable from the network access apparatus 20.

Regardless of whether the image and task selection are processedremotely or locally, the associated task associated with the object maybe stored as either a custom or a personalized task for future re-use orrepetition. Some customized or personalized task options may be sharedto other users via the server system 40. An option may be provided todefine and/or configure new task options by either the user or a serviceprovider. New task options may be made available immediately or delayedfor later use.

Flow of the method may be redirected back to block 10, wherein either animage of another scene is captured and processed based on a taskselection, or the same image is processed based on a different taskselection, to perform one or more other tasks (not specificallyillustrated).

FIG. 3 illustrates an embodiment of a hierarchy of object class and taskmenus. An object class menu 70 comprises a person class, a productclass, a company class, a location class and an event class.

The person class is used to perform a task associated with at least oneperson in the image. The person class may be selected if the imagecomprises a human face, for example.

The product class is used to perform a task associated with at least oneproduct in the image. The product may be in a visually recognizablepackage and/or container, the product itself may bevisually-recognizable, or the product may have a visually-decodableoptical code.

The company class is used to perform a task associated with a company,wherein at least one object in the image is associated with the company.The image may comprise a product either manufactured or sold by thecompany, a building which houses the company, of a company logo, forexample.

The location class is used to perform a task associated with a location,wherein at least one object in the image is associated with thelocation. The image may comprise a landmark (e.g. Eiffel tower or Statueof Liberty), or a building at the location, for example.

The event class is used to perform a task associated with an event,wherein the at least one object in the image is associated with theevent. The image may comprise an amusement park, a theme park, astadium, an arena, a theater or another event venue, for example.

In response to a selection of the person class from the object classmenu 70, a task menu 72 is provided. The task menu 72 provides a firstoption to get information about a person, a second option to contact aperson and a third option to arrange an event with a person. In responseto a selection of the second option, a task menu 74 is provided. Thetask menu 74 provides an option to send an e-mail to the person, anoption to send physical mail to the person, an option to place atelephone call to the person, and ah option to place a video call to theperson. In response to a selection of the third option, a task menu 76is provided. The task menu 76 provides an option to arrange a meetingwith the person, an option to arrange dinner or another meal with theperson, an option to arrange a date with the person, an option toarrange seeing a movie or attending another event with the person, andan option to be introduced to the person.

In response to a selection of the product class from the object classmenu 70, a task menu 80 is provided. The task menu 80 provides a firstoption to get information about a product, a second option to learn howto perform various tasks associated with a product, and a third optionto purchase the product. In response to a selection of the first option,a task menu 82 is provided. The task menu 82 provides an option to viewa demonstration of the product and an option to view a set of featuresof the product and comparable products.

In response to a selection of the second option, a task menu 84 isprovided. The task menu 84 provides an option to learn how to use theproduct, an option to learn how to repair the product, an option toreplenish a supply for the product, an option to return the product, anoption to order an accessory for the product, an option to register awarranty for the product, an option to insure the product, an option ofprovide feedback for the product, an option to dispose the product, anoption to contact customer service for the product and an option tocontact a manufacturer of the product.

In response to a selection, of the third option, a task menu 86 isprovided. The task menu 86 provides an option to purchase the productfrom a dealer, an option to purchase the product from another source, anoption to schedule delivery of the product being purchased, and anoption to arrange payment for the product being purchased. Other optionsmay include purchasing the product either as-is, with a discount, as apart of a service such as one available at www.mercator.com, or as partof an aggregate ordering service such as those described in any of theabove-identified related applications.

In response to a selection of the company class from the object classmenu 70, a task menu 90 is provided. The task menu 90 provides an optionto schedule service from a company, an option to get job employmentinformation for a company, an option to purchase stock in a company, anoption to provide feedback to a company and an option to pay an invoiceto a company. Each of the above tasks may be performed based on an imageof an object associated with the company.

In response to a selection of the location class from the object classmenu 70, a task menu 92 is provided. The task menu 92 provides an optionto arrange a visit to a location, an option to get historicalinformation about a location, an option to get at least one picture of alocation, an option to get a postcard of a location and an option to gettravel information to a location.

In response to a selection of the event class from the object class menu70, a task menu 94 is provided. The task menu 94 provides an option toobtain information associated with an event and an option to order aticket to an event. Information associated with an event may includescheduling information, food information, transportation information,and/or rides information (e.g. at an amusement or theme park), forexample. Various options to order a ticket include pre-ordering a ticketfor a later date (which may be discounted), or ordering a walk-in ticketfor the same day (which may be regularly-priced). Another example of anevent class task includes making reservations at a restaurant.

FIG. 4 illustrates an embodiment of the network access apparatus 20. Inthis embodiment, the network access apparatus 20 comprises a wirelesstelephone 100 comprising a speaker 102 and a microphone 104. The displaydevice 24 comprises a liquid crystal display on the wireless telephone100. The user input interface 32 comprises a telephone keypad 106 andany additional buttons (not specifically illustrated) on the wirelesstelephone 100.

The digital camera 16 is accessible either at an end of the wirelesstelephone 100 as illustrated, or at an alternative portion of thewireless telephone 100. For purposes of illustration an example,consider the digital camera 16 capturing an image of a depleted battery110 which the end user wishes to dispose.

After capturing the image, the display device 24 provides the objectclass menu 70. The end user selects the product class from the objectclass menu 70 using one or buttons on the wireless telephone 100.

In response to selecting the product class, the display device 24provides the task menu 80 as shown in FIG. 5. From the task menu 80, theend user selects the “learn how to . . . ” option using one or morebuttons on the wireless telephone 100.

In response to selecting the “learn how to . . . ” option, the displaydevice 24 provides a portion of the task menu 84 as shown in FIG. 6.Since the dispose task option is not displayed in the portion of thetask menu 84, the end user uses one of more buttons on the wirelesstelephone 100 to scroll down to another portion of the task menu 84, asshown in FIG. 7. The end user selects the “dispose” task option usingone of more buttons on the wireless telephone 100. In response thereto,the wireless telephone 100 receives battery disposal information from acomputer network resource (e.g. the computer network resource 61 in FIG.2). The battery disposal information is displayed by the display device24, as shown in FIG. 8. After reading the battery disposal information,the end user could schedule removal of the battery by navigating to anassociated task option using the wireless telephone 100.

Thus, there have been described herein several embodiments including apreferred embodiment of method and system for facilitating tasks usingimages and selections from object class and task menus.

It will be apparent to those skilled in the art that the disclosedinvention may be modified in numerous ways and may assume manyembodiments other than the preferred form specifically set out anddescribed above. For example, some of the acts described with referenceto FIG. 1 may be performed either concurrently or in an alternativeorder.

Accordingly, it is intended by the appended claims to cover allmodifications of the invention which fall within the true spirit andscope of the invention.

1. A method for use with a wireless communication device capable ofaccessing a computer network, the method comprising: receiving at awireless communication device and via the computer network, a sequenceof images of an event, at least one image of the sequence of imagesincluding a person; displaying an object class menu that includes atleast two of: a person class, a product class, an event class, alocation class and a company class; displaying a task menu including aplurality of tasks that are based on user selection of the event class;receiving via the server system, selection of a task option from theplurality of tasks, the task option including obtaining informationassociated with an event; and facilitating the task via communicationwith the computer network and based on the recognition of the person inthe at least one image via the server system, wherein facilitating thetask includes determining event information for the person.
 2. Themethod of claim 1 wherein the event information includes at least oneof: scheduling information, food information, transportation informationand ride information.
 3. The method of claim 1 wherein facilitating thetask further includes ordering a ticket.
 4. The method of claim 3wherein the ordering of the ticket includes at least one of:pre-ordering the ticket for a later date; and ordering the ticket for acurrent date.
 5. The method of claim 1 wherein facilitating the taskincludes sending an email to a person.
 6. The method of claim 1 whereinfacilitating the task includes placing a call to a person.
 7. The methodof claim 1 wherein at least one task option is configured as acustomized task.
 8. The method of claim 7 wherein the customized task isshared by other users of other wireless communication devices via aserver system.
 9. The method of claim 1 wherein the plurality of tasksinclude previously added task options made available from a serviceprovider.